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This paper reports on the development and refinement 
of the. Connecticut School Effectiveness Questionnaire (CSEQ)' and the 
Connecticut School Effectiveness Interview (CSEI), the primary data 
collection tools used in the Connecticut State Department of 
Education School Effectiveness Project. The primary purpose of^both 
the CSEI and the CSEQ is to determine the extent to which a school 
displays. seven school effectiveness characteristics: safe and orderly 
environment, clear school mission, instructional l.eadership, high 
expectations, opportunity to learn and student time oh task, frequent 
monitoring of student progress, and home-school relations. The CSEI 
is a 67-item instrument, with its items grouped by characteristic. 
Responses to each item are coded along a five-point descriptive 
continuiam. The CSEQ is a 100-item instrument, with content parallel 
to the CSEI. It is a paper and pencil technique in which the 
respondents respond to each item along a Likert-type scale. Items are 
not grouped by construct, but are randomly dispersed throughout the 
instrument. Reliability and validity studies indicate that the 
current forms of the CSEQ and the CSEI do provide accurate ajid 
consistent . data. Multitrait-multimethod analysis provides preliminary 
support for the identification of the seven school effectiveness 
characteristics as measurable constructs. (BW) 
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A Comparison of Interview and Questionnaire rechni(|tjes 
Used in' the Connecticut School Effectiveness Project 

A Report of Work in Progress 



r 1 t 

This paper reports on the development and refinement" of The Coylecticut 
School Effectiveness Questionnaire (CSEQ) ^nd ThP Connecticut SchQ^6l Effec- 
tiveness .Interview (CSEl) . These two Instruments are the primary d^ta collection 
tools used in the. Connecticut State Department of Education School Effectiveness 
Project. This Project involves both a model and a process thromih which the 
Connecticut State Department of Education assists schools in improving .their 
effectiveness (Gauthier. 1983). Improvement efforts are focused .on seven school- 
level, "alterable" characteristics that have been consistently, associated with 
high levels of student achievement for al 1 students. These characteristics are: 
■ - 1. Safe and Orderly Environment . There is an orderly purposeful atmosphere 
which is free from the threat of physical harm. However, the atmosphere is not 
oppressi've and Is conducive to teaching "and learning. 

2. Clear School Mission . There is a clearly-articulated mission of the 
school through which the staff shares an understanding of and a commitment to 
instructional goals, priorities, assessment procedures and eccountabi 1 i ty. 

3. Instructional Leadership . The principal acts as the instructional 
leader who effectively communicates the mission of the school to the staff, 
parents and students and who' understands and applies the characteristics of 
instructional effectiveness in the management of the instructional program of 
the school. 

4. Hi gh Expectations . The^school displays a climate of expectation in 
which the staff believes and ^demonstrates that students can attain mastery of 
basic ski lis- and that they (the staff) have the capabi 1 ity to help students 
achieve such mastery.. 



5. Opportunity to Learn anti Student Time on. Task . Teach&rs allocate 
a significant amount of classroom time to instruction in basic skill areas. 
For a high percentage of that allocated time students are engaged in planned. 

learning activities. 

6. Frequent Monitoring of Student Progress . Feedback, on student - 
academic progress is obta^ined frequently. Multiple assessment methods such 
as teacher-made tests, samples of student work, mastery ski 1 Is checklists, 
and criterion-referenced tests are used. The results of testing are iispd to 
improve individual student' performance and also to improve the instructional 
program. 

. 7. Home-School Relations . Parents understand and support the basic 
mission of the school and are made to feel that they have an important rore 
in achieving this mission. 

The primary purpose of both CSE^. end the CSEQ is to determine the extent to 
which a school displays the school effectiveness characteristics. A school 
faculty then uses data from the self-analysis to develop action plans for 

school impravement. 

A secondary, but related purpose"of" these instruments, is- to collect data 
necessary to transform the descriptive, qualitatively grdiinded characteristics 
into quantifiable operationally defined constructs. A complete description of 
the initial development, rationale, and research-base for" these instruments is 
available in works by Villanova (1982) and NEREX (1981). This paper will 
describe briefly initial reliability and validity information on both the C5E1 
and the CSEQ. Also, a preliminary comparison of data col lected with both 
instruments will be presented 'through a muTti tra^it-mu 1 timethod (MTMM) analysis 
(Campbell & Fiske. 1959) and through a comparison of data, col lected at the 
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school level in 10 schools 
INSTRUMENTS 

. Th^g^nnecticut ScIk ' :U veneSS Interview is a 67 item instrument 
designed to be used by a tr^ interviewer.. Items are grouped by character- 
istic and interviewees respond to each question. Interviewers are trained to 
code responses to each item a'ong a five point descriptive continuum. Re- 
sponses are summarized and pr »'~i led fop each school'. 

A preliminary analysis consisted of determining the internal, consistency 
of each of the scales and identifying poor jtems. Table 1. indicates the coeffi- 
cient alphas (Hull & Nie, 1981) of the current form of the CS& range from 
.66 to .93. Further item and scale reyisions-are presently taking place. 

TABLE I . 

Reliability Estimates for The Connecticut School Effectiveness Interview 

Number of ' Alpha Internal' . 

Items Reliabilities 



. Safe and Orderly Environment 
Clear School Mission 
Instructional Leadership 
Expectations 
Opportunity to Learn 
Monitoring Student Progress 
Home/School Relations 

Total N =r 423 . 

* The Connecticut School E ffectiven ess Questionnaire is a 100 i Lcm .instrument 
also designed to assess the extent to which the seven school effectiveness 
characteristics exist in a school. The content of the Questionnaire parallels 
that of the Interview. It is a Daper and penci T technique in which the respon- 
dent responds to each item' along a Uikert-type scale. Items are not grouped 



5 
11 

14 
10 
9 
8 
10 



.66 

.93 

.81 

.69 

.78. 

.86 

.66 
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'by construct, but are randomly dispersed throughout the instrument. 

Preliminary analyses of the current form of the CSEQ have included deter- 
mining the internal consistency of each, scale asV-well as a measure of stability. 
Table II reports the results of the analyses. - 

TABLE II-*, 

Reliability Estimates .for The Connecticut School Effectiveness Questionnaire 



^ . Number. of **Alpha Internal . *Test-Retest . 

Categories j^ems .. Reliabilities Reliabilities 



Safe arid Orderly Environment 15 .87 . .85 

Clear School Mission 14 .90 .90 

Lnstructional Leadership 25 .93 .83 

Expectations . 12 .55 .69 

Opportunity to Learn 12 .66 .74 

Monitoring Student Progress 12 .77 .67 

Home/School Relations . 10 .89 .82 



*rest-Retest N = 60, data coll-ected in one school 
**N = 423 

Item and scale data are currently being used to revise, adjJ, and delete 
items from the Expectations, Opportunity to Learn, and Monitoring scales. 

In an effort to establish construct validity, factor analysis has been used 
in both an exploratory and confirmatory mode. The match between hypothesized, 
a priori foe tors and derived factors (Marsh, Smith, & Barnes, 1984) was strong 
(coefficient alphas>.85) for the categories: Clear School Mission, Safe and 
Orderly Environment, Instructional Leadership, and HomeVSchool Relations. 
Results from these analyses wi 1 1 be reported in a future paper.. 

MULTITRAIT-MULTIMETHOD ANALYSIS ' ' . 

As the seven school effectiveness characteristics are assesse'd through 

' ■ ■ y ' i : 

two" distinctly different methods, investigation of the construct validity of 
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each of the seven school effectiveness measures can be = addressed through 
a multitrait-multimethod analysis (Campbell & Fiske, 1959; Marsh et al., 1984). 
School effectiveness is certainly a multifaceted concept. Evidence that 
the seven identified characteristics are in fact separate but interactive, 
constructs is derive_d primarily from the content validity established, 
through a review of the literature and an initial content validation study 
(Villanova, 1982).. Factor analytic studies may strengthen these findings. 
However, MTMM analysis can also demonstrate the multidimensionality of school, 
effectiveness. Convergent and discriminant validity information on each ^ 
hypothesized construct is' the chief result of the MTMM analysis. Table III . 
contains the data for such analysis. 

Convergent validity refers to a confirmation on the meaning of a trait ' 
measured by different methods. The more distinct the methods," the more 
convergent validity is established instead of a measure of reliability 
(See Campbell & Fiske, 1959, p. 84). "Discriminant validity refers Lo the 
distinctiveness of the var-ious traits, and .it is inferred from the relative 
lack of correlation among the different traits when compared to the convergence 
coefficients" (Marsh et al., 1984, p. 335). • 

Campbell and Fiske (1959). and Marsh et a.l .( 1984 ) suggest four specific 
criteria to be used to infer convergent and discriminant validity. In 
analyzing data presented in Table III the^ fol lowing inferences are niade: 

1. Convergent validities in the diagonal, of the heterotrait/heteromethod 
square (underlined) are suljstantlal. Evidence for convergent validity exists. 

2. Convergent validities are generally higher than correlations ob- 

tained between that variable and any other variable having neither trait 

* . " ' . <• 

or method in 'common. Even the relatively low convergent validities are 

greater than other correlations in the same row qr column. Monitoring Student 
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Hultitrait-Multimettiod Analysis: Seven School Effectiveness Variables Measured with the Connecticut School 
Effectiveness" questionnaire and The Connecticut' School^ Effectiveness Interview , 



(N = 247) 
Questionnaire 



Interview' 



Ai Bi -.Ci Dl El , fl Gl • ' A2 Be C2 fl k h k 



jstionnaire '. ' , 



Safe and Orderly Environment ky (.87) 

:iear School Mission Bi .67 (.90) 

Instructional Leadership Ci .51 .51 (.93) 

High Expectations ■" D] .44 .46 ,35 (.55)' , _ 

Opportunity to Learn Ei .67 .65 , .40 .49 (.66) 

Monitoring Student Progress Fi .54 .70 .45 • .45 .62 (.77) 

Home/School Relations Gi .72> .66 .36 .58 .59 .57- (.89) 



/ 



terview • <a 

Safe and 0ritfl?Enviro«i.A2 .75'! .48 . 38 , ' 

Clear School Hissior 62 .37. .| • • • • • ■ ■ ' ' , g,, ' 
mmimi\ Leadership Cj ".39 . . . f • . • ■ !• • 
High Expectations D2 . ■ ■ -f . • ? • ■ • • • ' , 
Op portity to Learr .£2 • • • -^^ • • • • •• 38 ' (SSl'- 
Iuorin, St« Progress F2 ■ • • . f f • ■ ; • ,; :§ 

Home/School Relations . ^ u2 -^^ " ° ' ■ • — , 



NOTE' Values in the main'diagonalare coefficient alpha reliabilities (Hull & flie,J981): Underlined values are 
■ ' convergent validities relating responses on the CSEQ to responses on the C3:I. 
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Progress (Fi, F2) is^a'notable exception. ^ . 

3. Generally, greater correlations^ti the. heterotrait, mohomethod 
triangle (upper i-eft) reveal more overlap and Interrelatedness among 
the characteristics measured with the CS^- Although, more than 50% of 
the correlations in the upper left triangular submatri^ are less than 

the four greatest convergent" validities. A degree of discriminant validity 

seems to be established. . 

There is generally a higher correlation between two difforont, met hods 
measuring the same trait than with measures designed to assess different 

.0 

traits using the same method. Discriminant validity appears stronger 
when convergent validities are compared tp -the Interview. heterotrait 

triangle (lower right) . - ' 

4. The pattern of correlations among questionnaire and interview 
mea.^res is similar in both monomethod triangles and the heteromethod block^. 

This preliminary analysis does provide significant support for both 

eonvergentN^d discriminant ^^alidity: Further' analysis, leads to' inferences 

related to the interrelationship among the seven factors to be explored. 

For example, the consistently high correlation'between Safe and. Orderly 

Environment (a'i. A2) and Home/School Relations (G1. G2) suggests the impact 

of the school and home together'. . Further studies followincr the next phase 

of rnstrument refinement will investigate the preliminary findings surfacing 

• if 

here. 



SCHOOL LEVEL COMPARl SONS,OF CSEQ AND CSE.l SUMMARIES 

- School Faculties participating in the School Effectiveness Project ^ , 
complete both the CSEl. anti the CSEQ. Summarized responses from both 
instruments at the school level provide the opportunity to compare and contrast 



'the data collected from both instruments. 

■ For this-analysis mean scores for each scale were created through 
assigning a number from 1 to 5 to each of the Likert-type response 
categories for both the Interview and Questionnaire. Categories or charac-' . 
teristics were then ranked, from 1 to 7 for both instruments in each school. 
The spearman rank-order correlation coefficient (Ps) was then used to - 
'.determine the degree of relationship between the ranks ofi the two instn- 
ments. Given.the importance of convergent validity for these measures, a 
positive and. statistically significant relationship between the two data sets 
was expected, Table IV indicates.' that.. in^six of ten schools the relationship 
between th| CSEQ ranks and the CSU ranks were significant.-^ 
. TABLE IV 

Spearman. Rank Order Coefficients for CSEI and CSEQ for Ten Schools. . 



School 


rs 


School / 




School A 
School B 
■School C 
r School D 
School E 


.68 
.86* . 
.73* 
.86*. . 
.29 


School F 
School G. 
" ^S'chool H 
" Schoo 1 1 
School J 


'l.O ** 
.70 
-.64 
.82* 
.90**- 



*P< .05 

**P< .001 . ■ 

,„ Appendex A means, standard deviation and r.anMngs are displayed for 
each o; the schools in this study. A low mean score In an area on hoth the 
interview and Questionnaire for a- given school is «ften^,e important alemenl 
in'' a school facultys- decision to design an action plan in that area in the ' 
school improvement process. A review of Table V indicates that- in each of . 



the 10 schools there is considerable overlap between the two highest 

priority areas as determined through .a simple ranking of mean scores. 

TABLE' V- ; ' 

Characteristics with the Lowest Mean Scores on the^CSEQ and the 

eSEI for Each School 



Ranked 
School 7th (CSEQ) 



Ranged 
7th (CSEI ) • 



Ranked 
6th (CSEQ) 



• Ranked 
6th (CSEI) 



A 

B 
C 
D 
E 
F 
G 
H 

I- 

J 



Home/School 
Home/School 
Home/School 
Home/School 
Home/School 



Lead. 

Exp. 

Lead.. 

Exp- 

Lead, 



\ 



Home/School 

Lead. 

Lead. 

Home/School 
•Home/School 
Lead. 
Lead; 
Lead. 
Lead. 
Lead- 

— 6^ 



^ Exp. 
Exp. 
Exp. 
Exp. 
ExR. 
St3 ' 

• Lead.. 

• Home /School 
Lead. 

Exp- 



Lead. 

SO 

SO 

Lead, 
Lead: 

^sd 

Exp. 
Exp. 
Exp-v 
.Exp. 



The schools in which the Spearman rank order coefficient did not reveal 
a statistically significant relationship jSchools A, E, G. H) do appear t6 
have considerable agreement in terms of characteristics most in need of 
improvement. This' observation ^along with the'significant rank order cor- ./ 
relations in six of the schools does provide strong support for both the . 
CSafaiid the csh as useful \latcV ccJI lection instruments. Simi lar priority 
areas do emerge from both the Interview and ,the Questionnaire. 



SUMMARY ■ ' > 

'. ■ ^ ^ 

This is presented as a report of work in^'progress. Reliability and 

validfty -studies indicate that^the current forms of the CSEQ and the CSn 

do provide accurate and consistent data. Further analyses do need to be • 



carried out as indicated. Revisions of both instruments with particular 
emphasis on "weaker" scales are necessary. MTf-lM analysis does provide 

preliminary support for the identification of the seven school effective- 

■ . ' o . ■ ■ 
ness characteristics as measurable constructs. The i^eview of comparative 

school level data also indicates that important and useful patterns do 

exist between data collected with the CSEQ and the CSEl . A more finalized 

report on these analyses and further developments (i.e., factor analytic ^ 

study) is expected to be presented within \the next year. -v 

■ - ' ' " \ 

■ ■ ■ - ' \. : 
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APPENDIX. A 



[Safe and Orderly Environment = SO; Clear School Mission = GSM; Instructional 
Leadership ='LEAD: Expections = EXP; Opportunity to Learn = OPP; Frequent 
Monitoring of Student Progress = MON; Home/School Relations =H/S] 



SCHOOL A 



CSE Questionnaire . ' CSE "Interview 

Rank M SD 





M 


SD 


Rank 


SO c. 


3.17 


.73 


5 


CSM 


3.68 


.47 


1 


LEAD 


3.22 


.50 


4 


EXP 


3.02 


.39 


6 


OPP 


3.29 


.39 


3 


HON 


3.58 


.36 


2 


H/S 


2.60 


.47 


7 



SCHOOL B 

CSE Questionnaire 





M 


SD 


Rank 


SO 


3.27 


.74 


4 


CSM 


3.64 


.39 


1 


LEAD 


3.05 


.70 . 


5 


EXP 


2.92 


.51 


6- 


OPP 


3.40 • 


.38 


3 


MON 


3.54 


.41 


2. 


H/S 


2.73 


.55 


7 



3 


3.66 


.47 


2 


3.96 


.bZ 


6 


3.27 


.62 


4 


^3.51 .. 


.49 


5 


3.45 


.51 


1 • 


4.11 


.65 


7 '■ 


3.03 


.54 




CSE Interview 




Rank 


M 


SD 


3 


3.43 


.59 


2 


3.51 


.82 


7 


2.53 


.59 


5 


3.04 


.44 


4 


3.44 


.53 


1 


3.53 


.50 


6 


2.96 


.52 



SCHOOL C 





CSE Questionnaire 
M SD 


Rank° 


Rank 


CSE Interview 
M 


SD 


SO 

CSM 

LEAD 

EXP 

OPP 

MON 

H/S 


3.04 

3.43 

3.08 

2.84 

3.30 , 

3.40 

2.37 


.75 
.69 
.56 
.49 
.45 
.58 
.58 


5 
1 
4 
6 
3 
2 
7 


4 
2 
7 
5 
3 
1 

6 


3.35 

3.89 

2.50 

2.96 

3.36 

3.96 . 

2.54 


.76 
.62 
.71 
.53 
.55 
.63 
.55 
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SCHOOL D 



CSE Questionnaire 





M 


SO 


3.31 


CSM 


3.70 


LEAD 


3.23 


EXP 


3.02 


OPP 


3.47 


MON 


3.74 


H/S 


2.67 



CSE Interview 



SD 


Rank 


Rank 


M 


SD 


.59 


4 


3. • 


. 3.24 


.51 


.49 


2 


4 


3.13 


.71 


.71 . 


5 


6 


2.50 


.60 


.39 


6 


5 


2.97 


.57 


.44 


3 


. 2 


3.46 


.48 


.46 


1 


1 


3.99 


. .47 


.44 


7 


7 


2.45 


.36- 



SCHOOL E 



CSE Questionnaire 





M 


SD 


R,ank 


SO 


3.81 


. .53 


5 


CSM 


3.99 


.43 


1 


LEAD • 


3.93 


.^54 


3 


EXP 


3.24 


.57' 


6 


OPP 


3.82 


.47 


4 


MON- 


3.99 


.47 


2 


H/S 


3.03 


.51 


7 



CSE Interview 



Rank 


M 


SD 


1 ■ 


4.02 


.37 


.4 


3.33 


.52 


6 


3.26 


^ .56 


5 


3.32 


.42 


2 


3.67 


' .46 


3 


3.65 


.55 


7 


3.08 


.40 



SCHOOL F 



CSE Questionnaire 





M 


SD 


Rank 


SO 


2.97 


.71 


6 


CSM 


3.41 


.58 


3 


LEAD : 


2.52 ■ 


.63 


7 


EXP 


3.09 


.35 


5 


OPP 


3.51 ■ 


.39 


2 


MON 


3.58, 


.■41 


1 


H/S 


3.18 


.59 . 


4 



CSE Interview 
Rank M 



6 
3 
7 
5 
2 
1 
4 



3.30 
3.35 
2.46 
3.26 
3.44 
3.63 
3.26 



SD 



.57 
1.01 
.72 
.50 
.48 
.61 
.40 



SCHOOL G 



CSE Questionnaire 



ERIC 





M 


SD 


so 


4.34 


.38 


CSM 


4.61 


.39 


LEAD 


3.71 - 


.59 


EXP 


3.68 


, .53 


OPP 


4.11 


.40 


MON 


4.18 


.48 


H/S 


4.26 


.41, 



Rank 



.2 
1 

6 
7 

5 
4 
.3 



15 



CSE 


Interview 




Rank 


M 


SD 


5 


, 4.31 


.29 


1 


4.90 


.15 


7 


3:44 


.56 


6 


4.10 ' 


.50 


4 


4.40 


.39 


2 


4.57 . 


.35 


3 


4.48 


■ .45 
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SCHOOL H 



CSE Questionnaire ' CSE Interview 





M 


SD 


SO 


3.68 . 


.38 


GSM 


3.66 


.31 


LEAD 


3.25 


.60 


EXP 


3.3% 


• .45 


OPP 


4.08 


.35 


MON 


3.79. 


.32 


H/S 


3.32 


.42 



Rank 


Rank 


M 


SD 


3 . -:• 


3 


3.95 


.48 


4 


1 


4.37 


.52 


7 




2.94 


.67 


5 


* 6 


3.57 


.42 


t 


4 


3.92 


.50 


2 


2 


4.29 


.43 


6 


5 


3.87 


.39 



SCHOOL I 

CSE Questionnaire CSE Interview 

Rank M S D 

2 3.93 .46 

3 3.84 .53 
7 2.83 .53 
6 3.24 .42 

.4 3.74 .41 

1 4.02 .50 

5 3.44 . .42 





M 


SD 


. Rank, 


SO 


3.92 


.41 


2 


CSM 


4.09 


.51 


1 


LEAD 


3.23 


.61 


6 


EXP 


3.. 14 


.46 


7 


OPP 


3.74 


.34 


4 


MON 


3.89 


.58 


3 


H/S- 


3.30 


.33 ^ 


5 



SCHOOL 'J 





CSE Questionnaire 






M 


SO 


Rank 


SO 


4.36 


.33 


1 


CSM 


4.26 


.45 


2 


LEAD . 


3.23 


.53 


7 


EXP 


3.33 


.38 


6 


OPP 


3.99 


.35 


4 


MON 


4.05 


.43 


3 


H/S 


' 3.84 


.41 


5 



CSE Interview 

Rank M SD 

1 4.56 .29 

4 4.12 .56 
7 2.69 .55 
6 3.77' .50 
3 4.16 .42 

2 4.17 .48 

5 3.92 .47 
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